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On  Achieving  Fairness  in  the  Joint  Allocation  of 
Processing  and  Bandwidth  Resources:  Principles 

and  Algorithms 

Yunkai  Zhou  and  Harish  Sethu 


Abstract —  The  problem  of  achieving  fairness  in  the  allocation 
of  the  bandwidth  resource  on  a  link  shared  by  multiple  flows  of 
traffic  has  been  extensively  researched  over  the  last  decade.  How¬ 
ever,  with  the  increasing  pervasiveness  of  optical  networking  and 
the  occasional  trend  toward  using  over-provisioning  as  the  solu¬ 
tion  to  bandwidth  congestion,  a  router’s  processor  also  becomes 
a  critical  resource  to  which,  ideally  speaking,  all  competing  flows 
should  have  fair  access.  For  example,  if  the  network  is  not  fair 
in  allocating  processing  resources,  denial  of  service  attacks  based 
on  an  excessive  use  of  the  router  processor  (such  as  by  using  un¬ 
necessary  optional  headers)  become  possible.  In  this  report,  we 
investigate  the  issue  of  achieving  fairness  in  the  joint  allocation  of 
the  processing  and  bandwidth  resources.  We  first  present  a  simple 
but  powerful  general  principle  for  defining  fairness  in  such  sys¬ 
tems  based  on  any  of  the  classic  notions  of  fairness  such  as  max- 
min  fairness,  proportional  fairness  and  utility  max-min  fairness 
defined  for  a  single  resource.  We  apply  our  principle  to  a  system 
with  a  shared  processor  and  a  shared  link  with  max-min  fairness 
as  the  desired  goal.  We  then  propose  a  practical  and  provably  fair 
packet-by-packet  algorithm  for  the  joint  allocation  of  processing 
and  bandwidth  resources.  We  demonstrate  the  fairness  achieved 
by  our  algorithm  through  simulation  results  using  both  synthetic 
and  real  gateway  traffic  traces.  The  principies  and  the  aigorithm 
detailed  in  this  report  may  also  be  applied  in  the  allocation  of  other 
kinds  of  resources  such  as  power,  a  critical  resource  in  mobile  sys¬ 
tems. 

Index  Terms — Fairness,  resource  allocation,  processor  sharing, 
max-min. 


I.  Introduction 
A.  Introduction  and  Motivation 

Fairness  in  the  allocation  of  resources  in  a  network  shared 
amongst  multiple  users  is  not  only  an  intuitively  desirable  goal 
but  also  one  with  many  practical  benefits.  Fairness  in  traf¬ 
fic  management  can  improve  flow  and  user  isolation,  offer 
a  more  predictable  performance,  and  eliminate  certain  kinds 
of  bottlenecks.  In  addition,  strategies  and  algorithms  for  fair 
management  of  network  traffic  can  serve  as  a  critical  compo¬ 
nent  of  Quality-of-Service  (QoS)  mechanisms  to  achieve  cer¬ 
tain  guaranteed  services  such  as  delay  bounds  and  minimum 
bandwidths.  Fair  resource  allocation  strategies  can  also  help 
in  countering  certain  kinds  of  denial-of-service  (DoS)  attacks 
[1].  Various  formal  notions  of  fairness  have  been  proposed  in 
the  literature  to  precisely  define  what  is  fair  in  the  allocation 
of  a  resource  amongst  competing  flows.  These  include,  among 

This  work  was  supported  in  part  by  NSF  CAREER  Award  CCR-9984161  and 
U.S.  Air  Force  Contract  F30602-00-2-0501.  A  preliminary  version  of  this  re¬ 
search  appeared  in  Proc.  Int’l  Workshop  on  Quality  of  Service  (IWQoS),  Mon¬ 
terey,  CA,  June  2003. 


Others,  max-min  fairness  [2-5],  proportional  fairness  [6],  and 
utility  max-min  fairness  [7]. 

Based  on  these  notions  of  fairness — most  commonly,  based 
on  the  notion  of  max-min  fairness — much  research  over  the  last 
decade  or  two  has  focused  on  the  allocation  of  the  bandwidth 
resource  on  a  link  [3,4, 8-12].  It  has  also  been  shown  that  con¬ 
cepts  and  algorithms  for  achieving  fairness  in  the  allocation  of 
a  single  resource  can  be  extended  to  the  case  with  multiple  re¬ 
sources  of  the  same  kind  [13].  However,  as  flows  of  traffic 
traverse  a  computer  network,  they  share  many  different  kinds 
of  resources  such  as  link  bandwidth,  buffer  space,  time  on  the 
router  processors  and  also  electrical  power,  a  critical  resource 
in  mobile  systems.  The  ultimate  goal,  therefore,  should  be  the 
overall  fairness  in  the  joint  allocation  of  all  resources  shared  by 
the  flows  of  traffic  and  not  just  one  specific  kind  of  resource 
such  as  the  link  bandwidth.  For  example,  if  the  network  is  not 
fair  in  allocating  processing  resources,  DoS  attacks  based  on  an 
excessive  use  of  the  router  processor  (such  as  by  using  unnec¬ 
essary  optional  headers)  become  possible. 

The  significance  of  considering  the  fair  allocation  of  more 
than  just  the  link  bandwidth  is  increasingly  becoming  apparent 
today,  since  the  link  bandwidth  is  often  not  the  only  critical  re¬ 
source.  With  the  current  pervasiveness  of  optical  networking  in 
the  Internet  backbone,  and  with  the  occasional  trend  toward  us¬ 
ing  over-provisioning  as  the  solution  to  congestion  in  the  edge 
networks,  a  router’s  processor  is  often  also  a  critical  resource  to 
which,  ideally  speaking,  all  competing  flows  should  have  fair 
access.  Given  the  fact  that  processing  requirements  of  differ¬ 
ent  packets  vary  widely,  the  issue  of  fairness  in  the  allocation 
of  the  processing  resources  gains  significance.  In  addition,  be¬ 
sides  the  fact  that  packet  lengths  can  vary  widely,  the  presence 
of  optional  headers  and  the  various  kinds  of  control  informa¬ 
tion  carried  by  packets  create  a  wide  variation  in  the  ratio  of  a 
packet’s  demand  for  bandwidth  and  its  demand  for  processing 
cycles.  Thus,  packets  of  the  same  length  cannot  be  guaranteed 
to  have  similar  requirements  for  the  processing  resources  on  a 
router.  In  fact,  the  processing  delay  plotted  as  a  function  of  the 
packet  length  shows  that  the  processing  requirements  of  pack¬ 
ets  vary  across  a  wide  range  even  for  packets  of  the  same  length 
[14].  Thus,  one  cannot  achieve  overall  fairness  merely  with  the 
fair  allocation  of  link  bandwidth  alone,  or  merely  through  the 
fair  allocation  of  the  processing  resource  alone,  since  different 
flows — and  different  packets  within  the  same  flow — may  have 
very  different  demands  for  these  two  kinds  of  resources.  All 
of  this  begs  the  question  this  report  seeks  to  address;  how  does 
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one  achieve  fairness  in  the  joint  allocation  of  the  processing  and 
bandwidth  resources? 

The  need  for  fairness  in  the  joint  allocation  of  multiple  het¬ 
erogeneous  resources  has  also  been  recognized  in  other  con¬ 
texts  besides  the  one  discussed  here.  For  example,  it  has  been 
recognized  that  fair  allocation  of  both  the  channel  bandwidth 
and  the  power  consumed  needs  to  be  achieved  simultaneously 
in  mobile  networks  where  power  and  bandwidth  are  both  criti¬ 
cally  important  and  scarce  resources  [15].  However,  a  rigorous 
theoretical  framework  that  may  be  universally  employed  as  a 
guide  in  the  design  of  practical  algorithmic  strategies  for  the 
joint  allocation  of  such  heterogeneous  sets  of  resources  does 
not  exist. 

In  this  technical  report,  we  investigate  the  issue  of  fairness 
in  such  systems  and  develop  a  general  principle  that  forms  the 
foundation  for  the  design  of  practical  and  fair  strategies  for  use 
in  routers.  We  also  present  an  evaluation  of  the  practical  strate¬ 
gies  proposed  in  this  report  using  both  synthetic  and  real  gate¬ 
way  traffic  traces. 

B.  Contributions 

In  the  joint  allocation  of  the  processing  and  bandwidth  re¬ 
sources,  if  a  certain  resource  is  never  the  bottleneck,  then  the 
fair  allocation  strategy  degenerates  to  the  fair  allocation  of  just 
the  other  resource.  For  example,  if  the  available  bandwidth  is 
large  enough  that  no  flow  experiences  congestion  due  to  lack  of 
bandwidth  alone,  one  only  needs  to  worry  about  the  allocation 
of  the  processing  resource.  Fair  allocation  of  a  single  bottle¬ 
neck  resource  has  been  studied  extensively  in  the  literature  and 
has  led  to  a  large  number  of  practical  algorithms  that  are  in  use 
today  in  Internet  routers,  operating  systems,  and  transport-level 
protocols.  This  report,  on  the  other  hand,  answers  the  question 
of  what  is  a  fair  allocation  when  more  than  one  resource  is  con¬ 
gested  and  extends  the  notions  of  fairness  applied  to  a  single 
resource  to  systems  with  multiple  heterogeneous  resources. 

We  define  an  essential  resource  as  one  for  which  a  flow’s  de¬ 
mand  does  not  reduce  with  an  increase  in  the  allocation  of  other 
resources  to  the  flow.  A  number  of  resources  such  as  the  link 
bandwidth,  processor  or  power,  in  most  contexts,  are  essential 
resources.  On  the  other  hand,  buffer  resources  in  a  network  are 
often  non-essential  resources  as  per  the  above  definition;  for 
example,  in  a  system  with  a  buffer  and  a  link,  a  flow  uses  the 
buffer  only  if  the  link  resource  is  currently  unavailable  to  it,  and 
thus  a  flow’s  demand  for  the  buffer  resource  reduces  as  more  of 
the  link  bandwidth  is  allocated  to  it.  Note  that  a  non-essential 
resource  does  not  necessarily  mean  that  it  is  not  useful.  In  the 
system  model  used  in  this  report,  we  assume  that  the  flows  are 
in  competition  for  resources  that  are  all  essential.  The  issue  of 
achieving  fairness  in  a  system  where  flows  have  to  compete  for 
a  non-essential  resource  such  as  a  buffer  entails  a  different  set 
of  challenges  than  those  considered  here,  and  is  addressed  in 
some  other  recent  works  such  as  [16]. 

We  define  a  pair  of  resources  as  related  to  each  other  if  a 
flow’s  demand  for  one  resource  uniquely  determines  its  demand 
for  the  other  resource.  Resources  in  a  set  are  said  to  be  related 
if  each  resource  is  related  to  every  other  resource  in  the  set. 
Resources  in  real  scenarios  are  almost  always  related  since  the 
demands  of  a  flow  for  different  individual  resources  are  often 


related  to  each  other.  For  example,  since  each  packet  is  asso¬ 
ciated  with  certain  processing  and  bandwidth  requirements,  a 
specific  increase  in  a  flow’s  demand  for  link  bandwidth  is  typ¬ 
ically  associated  with  a  specific  increase  in  its  demand  for  the 
processing  resource.  A  simpler  example,  involving  multiple  re¬ 
sources  of  the  same  kind,  is  a  tandem  network  with  multiple 
links  where  the  demand  of  a  flow  for  bandwidth  is  the  same  on 
all  the  links.  In  the  system  model  used  in  this  report,  we  as¬ 
sume  multiple  resources  that  are  related,  although  we  make  no 
assumptions  on  the  specific  nature  of  the  relationship  between 
a  flow’s  demand  for  different  resources.  The  existence  of  a  re¬ 
lationship  between  the  demands  of  a  flow  for  various  resources 
calls  for  the  joint  allocation  of  these  resources,  as  opposed  to  an 
independent  and  separate  allocation  of  the  resources. 

The  primary  contribution  of  this  report  is  a  theoretical  frame¬ 
work  based  on  which  one  can  define  fairness  in  the  joint  alloca¬ 
tion  of  multiple  heterogeneous  resources  that  are  essential  and 
related.  We  make  no  assumptions  on  the  notion  of  fairness;  in 
fact,  our  framework  may  be  applied  to  any  of  several  notions 
of  fairness  such  as  max-min  fairness,  proportional  fairness  or 
utility  max-min.  Through  illustrative  examples,  we  claim  that, 
at  each  instant  of  time,  it  is  the  maximum  of  a  flow’s  normal¬ 
ized  demand  for  the  various  resources  that  should  count  in  the 
decisions  made  by  a  fair  resource  allocation  algorithm.  We  then 
develop  the  fundamental  principles  of  fairness  for  systems  with 
multiple  essential  and  related  heterogeneous  resources,  and  pro¬ 
pose  the  Principle  of  Fair  Essential  Resource  Allocation  or  the 
FERA  principle,  expressed  within  a  rigorous  theoretical  frame¬ 
work.  We  also  prove  that,  under  certain  conditions,  there  exists 
a  unique,  fair,  and  work-conserving  resource  allocation  policy 
which  satisfies  the  FERA  principle. 

Given  the  FERA  principle,  we  proceed  to  apply  it  to  a  system 
with  a  shared  processor  and  a  shared  link,  using  max-min  fair¬ 
ness  as  the  notion  of  fairness.  We  propose  an  ideally  fair  pol¬ 
icy,  called  the  Fluid-flow  Processor  and  Link  Sharing  (FPLS) 
algorithm,  for  the  joint  allocation  of  processing  and  bandwidth 
resources.  We  then  develop  a  practical  and provably  fair  packet- 
by-packet  approximation  of  the  EPLS  algorithm,  called  Packet- 
by-packet  Processor  and  Link  Sharing  (PELS).  The  PPLS  algo¬ 
rithm,  based  on  an  extension  of  the  Deficit  Round  Robin  algo¬ 
rithm  [10],  has  a  per-packet  work  complexity  of  0(1).  We  il¬ 
lustrate  the  fairness  of  the  PPLS  algorithm  using  both  synthetic 
traffic  and  real  gateway  traffic  traces. 

Even  though  this  report  primarily  focuses  on  processing  and 
bandwidth  resources,  the  EERA  principle  may  be  readily  ap¬ 
plied  to  a  variety  of  contexts  beyond  those  discussed  in  this 
report. 

C.  Organization 

The  rest  of  this  report  is  organized  as  follows.  Section  II 
introduces  a  generic  notation  to  represent  notions  of  fairness. 
This  section  also  describes  the  general  system  model  with  mul¬ 
tiple  shared  resources  considered  in  this  study,  along  with  our 
notation.  Section  III  presents  the  Principle  of  Pair  Essential 
Resource  Allocation  for  the  system  model  under  consideration. 
Section  IV  applies  the  EERA  principle  to  a  system  with  a  shared 
processor  and  a  shared  link,  and  proposes  a  practical  and  fair 
scheduling  algorithm  for  the  joint  allocation  of  the  processing 
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and  bandwidth  resources,  called  the  Packet-by-packet  Proces¬ 
sor  and  Link  Sharing  (PPLS)  policy.  The  fairness  properties  of 
the  PPLS  strategy  are  demonstrated  by  simulation  experiments 
using  both  synthetic  and  real  gateway  traffic  in  Section  V.  Fi¬ 
nally,  Section  VI  concludes  the  report. 


Flow  1 
Flow  2 
Flow  N 


Fig.  1 .  A  general  system  model. 


11.  System  Model  AND  Notation 
A.  Generic  Notation  for  Notions  of  Fairness 

Consider  a  set  of  N  flows,  1  <  i  <  N,  competing  for  a  single 
shared  resource  which  may  be  consumed  at  a  peak  rate  of  R. 
Denote  by  Wi  the  weight  of  flow  i,  indicating  the  flow’s  relative 
rightful  share  of  the  resources.  For  a  flow  under  a  Differentiated 
Services  (DiffServ)  framework  [17],  its  weight  is  determined 
by  its  traffic  class  among  the  64  possible  classes;  for  a  flow  in 
a  best-effort  network,  its  weight  is  typically  the  same  as  that  of 
all  other  flows. 

Several  different  notions  of  fairness  have  been  proposed  in 
the  research  literature  for  the  allocation  of  a  single  shared  re¬ 
source  among  a  set  of  requesting  entities.  All  of  these  notions 
specify  a  particular  rate  of  consumption  of  the  resource  for  each 
of  the  flows,  given  the  consumption  rate  demanded  by  the  flows. 
In  this  subsection,  we  develop  a  generic  notation  that  can  ex¬ 
press  any  of  these  notions  of  fairness. 

Without  loss  of  generality,  we  assume  that  the  entities  com¬ 
peting  for  the  single  shared  resource  are  traffic  flows.  Let  di 
be  the  demand  of  flow  i  for  the  shared  resource.  Define  the 
normalized  demand  of  flow  i,  Di,  for  the  resource  as  follows: 


The  normalized  demand  of  flow  i  indicates  the  fractional  share 
of  the  resource  demanded  by  the  flow.  Denote  by  ai  the  allo¬ 
cated  resource  consumption  rate  for  flow  i.  Define  the  normal¬ 
ized  allocation  of  flow  i,  Ai,  as  follows: 


The  normalized  allocation  of  flow  i  indicates  the  fractional 
share  of  the  resource  allocated  to  flow  i.  Any  notion  of  fair¬ 
ness,  thus,  specifies  how  to  distribute  the  fractional  share  of  the 
resource  allocated  to  each  flow,  given  the  desired  share  of  this 
resource. 

For  the  sake  of  convenience,  throughout  this  report  we  use 
vectors  to  indicate  values  corresponding  to  a  set  of  flows.  We 
denote  a  vector  by  the  indexed  value  in  a  pair  of  square  brackets. 
For  instance,  we  denote  the  vector  of  normalized  demands  as 

m 

Therefore,  given  the  normalized  demand  vector  \Di\  and  the 
weight  vector  [zui],  any  given  notion  of  fairness  may  be  repre¬ 
sented  as 

[A,]=T{C,[Di\,[wi\)  (1) 

where  C  is  the  constraint,  described  later  in  greater  detail,  im¬ 
posed  on  the  system.  The  function  T  is  different  for  different 
notions  of  fairness  such  as  max-min  fairness,  proportional  fair¬ 
ness  or  utility  max-min  fairness.  Note  that  the  notion  of  fairness 
in  (1)  imposes  no  dimension  on  any  variable,  thus  making  it  ap¬ 
plicable  to  systems  with  multiple  heterogeneous  resources. 


Often,  a  fairness  notion  implies  a  certain  way  to  compute  the 
utility  to  a  flow  of  the  allocations.  Different  notions  of  fairness 
assume  different  utility  functions,  though  all  are  non-decreasing 
functions  with  respect  to  quantity  of  the  allocated  resource.  For 
example,  max-min  fairness  implies  a  linear  utility  function,  pro¬ 
portional  fairness  uses  a  logarithmic  utility  function,  and  in  util¬ 
ity  max-min,  each  flow  determines  its  own  utility  function.  The 
notion  of  fairness  in  (1)  represents  a  general  notation  to  describe 
how,  given  a  certain  vector  of  demands,  one  may  determine  the 
allocated  consumption  rate  of  the  resource  for  each  flow,  in  or¬ 
der  that  the  utilities  corresponding  to  the  allocations  satisfy  the 
given  fairness  notion  with  respect  to  the  demands  for  utility.  In 
other  words,  the  notation  of  (1)  implicitly  incorporates  utility 
functions  into  the  notion  of  fairness. 

The  constraint  C  is  used  as  a  parameter  in  the  function  T 
because,  given  the  same  demand  and  weight  vector,  the  fair  al¬ 
location  is  different  under  different  constraints  imposed  on  the 
system.  The  constraint  C  can  be  used  to  incorporate  the  perfor¬ 
mance  level  achieved  by  the  allocation.  For  example,  an  allo¬ 
cation  of  no  resource  to  any  flow  may  also  be  considered  a  fair 
allocation  by  the  max-min  fair  criterion  albeit  one  that  leads  to 
very  poor  performance.  In  general,  this  parameter  allows  us  to 
define  the  fairness  of  non-work-conserving  allocation  strategies 
by  not  imposing  a  specific  level  of  performance  achieved  by  the 
allocation  in  the  definition  of  fairness.  As  a  simple  example,  the 
constraint  C  can  be  just  the  sum  of  the  utilities  achieved  by  all 
flows. 

Note  that,  in  the  research  literature,  notions  of  fairness  have 
not  been  defined  for  multiple  heterogeneous  resources^.  We 
use  the  above  notation  that  specifies  a  notion  of  fairness  for  a 
single  resource  and  extend  the  notion  to  multiple  heterogeneous 
resources  in  subsequent  sections. 

B.  System  Model  and  Assumptions 

In  our  system  model,  a  set  of  N  flows,  1  <  z  <  A^,  compete 
for  a  set  of  K  related  and  essential  resources,  1  <  j  <  AT,  as 
shown  in  Fig.  1.  As  also  described  in  Section  I-B,  we  define  an 
essential  resource  as  one  for  which  a  flow’s  demand  does  not 
reduce  with  an  increase  in  the  allocation  of  other  resources  to 
it.  Since  a  buffer  is  often  not  an  essential  resource,  our  assump¬ 
tion  that  flows  only  compete  for  essential  resources  implies  that 
if  there  are  buffers  in  the  network  shared  by  the  flows,  these 
buffers  are  of  infinite  capacity  so  that  the  flows  never  compete 

^  Some  notions  of  fairness  such  as  max-min  fairness  and  proportional  fairness 
can  be  defined  for  multiple  resources  of  the  same  kind  (e.g.,  a  network  of  links), 
under  the  assumption  that,  if  a  flow  receives  allocation  of  several  resources,  the 
allocation  of  each  of  these  resources  it  receives  is  identical  [2,6].  However,  it  is 
not  straightforward  to  extend  these  notions  of  fairness  to  systems  with  multiple 
heterogeneous  resources.  On  the  other  hand,  it  can  be  readily  verified  that  our 
framework  is  the  same  as  these  notions  of  fairness  if  the  shared  resources  are  of 
the  same  kind. 
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for  the  buffer  resource.  In  developing  our  fundamental  princi¬ 
ples  of  fairness,  we  make  no  assumptions  on  the  specific  actions 
of  the  scheduler  or  the  specific  order  in  which  the  packets  use 
the  K  resources. 

Note  that  in  this  general  model,  we  also  make  no  assumptions 
on  the  internal  architecture  of  the  set  of  shared  resources.  It  can 
be  a  simple  sequence  of  resources  such  as  in  a  tandem  network 
with  multiple  links,  a  parallel  structure  such  as  the  resources  of 
electric  power  and  bandwidth  in  a  wireless  sensor  network,  or 
a  more  complex  hybrid. 

Denote  by  Rj  the  peak  rate  at  which  resource  j  may  be  con¬ 
sumed.  For  example,  in  the  case  of  a  link  resource  L,  is  the 
peak  bandwidth  available  on  the  link.  As  before,  denote  by  Wi 
the  weight  of  flow  i.  Let  dij  be  the  consumption  rate  demanded 
by  flow  i  for  the  shared  resource  j.  Our  assumption  of  related 
resources  implies  that,  given  di^k,  one  can  determine  dij  for 
all  j  7^  k.  Denote  by  of  p  the  consumption  rate  of  the  shared 
resource  j  allocated  to  flow  i  under  the  allocation  policy  q. 

III.  The  FERA  Principle 
A.  The  Concept  of  the  Prime  Resource 

We  begin  with  a  few  preliminary  definitions. 

Definition  1:  Define  the  normalized  demand  of  flow  i  for 
resource  j,  Dij,  as  follows: 


Define  the  largest  normalized  demand  of  flow  i,  Di,  as  the 
maximum  amongst  the  normalized  demands  of  flow  i  for  all 
resources.  That  is. 


Di  =  max  {Di  j}  . 
j 

Definition  2:  Define  AJ  ^  as  the  normalized  allocation  of 
resource  j  to  flow  i  under  allocation  policy  q,  i.e.. 


Definition  3:  The  largest  normalized  allocation  of  a  flow 
i  under  allocation  policy  q,  denoted  by  Af,  is  defined  as  the 
maximum  amongst  the  normalized  allocations  to  flow  i  of  all 
resources.  That  is, 

Al  =  mAx{Al.}  . 

Definition  4:  Under  an  allocation  policy  q,  a  resource  is 
said  to  be  a  prime  resource  of  flow  i,  denoted  by  if  and 
only  if,  the  normalized  allocation  of  this  resource  to  flow  i  is 
the  largest  normalized  allocation.  In  other  words. 


B'^  =  arg  max  j  }  =  arg  max  ■ 


where  arg  max^;  f{x)  indicates  the  value  of  the  argument  x  cor¬ 
responding  to  the  maximum  value  of  function  f{x).  In  other 
words,  we  have 

=  max{AU}  =  Af 


TABLE  I 

Examples  illustrating  what  is  a  fair  allocation  in  a  system 

WITH  A  SHARED  PROCESSOR  P  AND  A  SHARED  LINK  L.  IN  ALL  OF  THESE 
EXAMPLES,  THE  TOTAL  AMOUNT  OF  RESOURCE  P  IS  100  MHZ,  AND  THE 
TOTAL  AMOUNT  OF  RESOURCE  L  IS  100  MBPS. 


Flow 

Demand 

Allocation 

ID 

P  (MHz) 

L  (Mbps) 

P  (MHz) 

L  (Mbps) 

A 

1 

75 

25 

75 

25 

A 

2 

25 

75 

25 

75 

B 

1 

225 

75 

75 

25 

2 

50 

150 

25 

75 

1 

100 

20 

50 

10 

C 

2 

100 

10 

50 
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In  networking  terminology,  a  bottleneck  resource  is  one  that 
is  the  most  congested.  It  is  critical  to  note  that  neither  the  re¬ 
source  for  which  a  flow  has  the  largest  normalized  demand  nor 
its  prime  resource  under  an  allocation  policy  is  necessarily  the 
same  as  the  bottleneck  resource  in  the  system. 

Note  that  a  flow  may  have  more  than  one  prime  resource. 
The  prime  resource  is  defined  based  on  the  actual  allocations 
and  not  on  the  demand  of  the  flows  for  the  resources. 


B.  The  FERA  Principle 

We  introduce  our  principle  with  a  few  illustrative  examples 
shown  in  Table  1.  In  these  examples,  two  flows  with  equal 
weights,  labeled  as  1  and  2,  share  two  different  resources:  a 
processor  P  for  packet  processing,  and  a  link  L  for  packet  trans¬ 
mission.  The  system  model  in  these  examples  is  the  same  as  the 
one  we  will  discuss  later  in  Fig.  2.  The  peak  processing  rate  is 
100  million  processor  cycles  per  second,  i.e.,  100  MHz,  and  the 
peak  link  rate  is  100  Mbps.  Let  us  assume  linear  utility  func¬ 
tions  and  max-min  as  the  notion  of  fairness.  In  addition,  for 
the  sake  of  convenience,  we  also  assume  in  these  examples  a 
proportional  relationship  between  a  flow’s  demands  for  these 
resources,  and  therefore,  a  proportional  relationship  between 
the  allocations.  In  other  words,  the  ratio  of  a  flow’s  demand  for 
one  resource  and  its  demand  for  another  resource  is  always  a 
constant. 

In  example  A,  assume  that  packets  in  flow  1  are  all  small, 
and  therefore,  its  demand  for  bandwidth  is  small  relative  to  its 
demand  for  processing  time.  In  contrast,  assume  that  packets 
in  flow  2  are  large,  and  therefore,  its  demand  for  bandwidth  is 
large  relative  to  its  demand  for  processing  time.  To  better  il¬ 
lustrate  the  concept,  we  exaggerate  the  difference  between  their 
demands  as  follows:  flow  1  has  a  demand  of  75  MHz  for  pro¬ 
cessing  time  and  25  Mbps  for  bandwidth,  while  flow  2’s  de¬ 
mands  are,  respectively,  25  MHz  and  75  Mbps.  If  a  work- 
conserving  allocation  policy  is  used,  there  is  enough  of  both 
resources  to  satisfy  the  demands  of  both  the  flows  and  so  the 
allocations  are  exactly  the  same  as  the  demands  for  each  of  the 
resources.  Note  that  for  flow  1,  the  prime  resource  is  P,  while 
for  flow  2,  it  is  L. 
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Next,  consider  what  happens  when  both  flows  proportionally 
increase  their  demands  for  both  resources.  In  example  B,  in 
comparison  to  example  A,  flow  1  increases  its  demands  by  a 
factor  of  three  while  flow  2  doubles  its  demands.  Specifically, 
the  demands  for  flow  1  become  225  MHz  for  P  and  75  Mbps 
for  L,  while  those  for  flow  2  become  50  MHz  and  150  Mbps, 
respectively.  A  fundamental  principle  behind  the  max-min  no¬ 
tion  of  fairness  is  that,  given  no  additional  resources,  a  flow 
should  not  be  able  to  increase  its  allocation  by  merely  demand¬ 
ing  more.  Thus,  the  fair  allocation  should  be  the  same  as  in 
example  A,  as  shown  in  example  B.  Again,  the  prime  resource 
for  either  flow  remains  the  same  as  in  the  previous  example. 

We  discuss  example  B  further.  Obviously,  in  this  case,  nei¬ 
ther  flow  is  satisfied  by  the  allocated  resources.  Is  the  allocation 
actually  fair? 

One  might  argue  that  both  flows  should  get  equal  bandwidth 
from  a  fair  allocation,  since  ultimately  both  flows  will  depart 
from  this  system  and  the  processor  is  only  an  intermediate  re¬ 
source  before  the  flow’s  packets  reach  the  link  resource.  Based 
on  this  notion,  we  can  compute  the  allocations  as 

3x  +  x/3  <  100 

2x  <  100 

where  x  is  the  bandwidth  allocated  to  either  flow,  in  units  of 
Mbps.  It  can  be  readily  verified  that,  under  a  work-conserving 
allocation  policy,  flow  1  gets  90  MHz  of  processing  time  and 
flow  2  gets  only  10  MHz,  while  both  flows  get  30  Mbps  of  band¬ 
width.  While  this  allocation  underutilizes  the  link  resources, 
that  is  not  an  argument  against  its  fairness.  The  unfairness 
arises  from  the  fact  that  it  unnecessarily  favors  the  flow  whose 
prime  resource  is  the  “intermediate”  resource,  which  turns  out 
to  be  flow  1  in  this  case.  Another  argument  against  this  notion 
is  that,  even  though  it  is  true  that  the  processor  in  this  case  is  po¬ 
sitioned  ahead  of  the  link,  it  does  not  necessarily  mean  that  the 
processing  resource  becomes  less  important,  or  less  preferred, 
as  compared  to  the  link,  which  is  positioned  as  the  “final”  re¬ 
source. 

Another  allocation  philosophy  may  be  to  allocate  resources 
based  on  a  fair  allocation  of  the  most  congested  resource  as 
measured  by  the  sum  of  the  normalized  demands  for  the  re¬ 
source.  In  this  example,  the  processing  resource  P  is  the  most 
congested  resource.  One  may  allocate  resource  P  fairly  as 

2y  <  100 
y/3  +  3y  <  100 

where  y  is  the  processing  resources  allocated  to  either  flow,  in 
units  of  MHz.  Under  a  work-conserving  allocation  policy,  flow 
2  gets  90  Mbps  of  bandwidth  and  flow  1  gets  only  10  Mbps, 
while  both  flows  get  30  MHz  of  processing  resources.  Note 
that  this  allocation  philosophy  has  a  similar  weakness  as  the  one 
based  on  the  fair  allocation  of  the  link  resource.  It  unnecessarily 
favors  the  flow  whose  largest  normalized  demand  is  not  for  the 
most  congested  resource. 

One  may  suggest  the  following  slight  modification  to  the  al¬ 
location  strategy:  to  fairly  allocate  the  most  congested  resource 
as  measured  by  the  sum  of  the  normalized  allocations  for  the 
resource.  However,  it  can  be  shown  that  such  an  allocation  may 


not  exist.  Assume  a  certain  resource  r  is  the  most  congested 
resource.  Let  a  denote  the  flow  with  the  smaller  demand  for 
resource  r  and  let  /3  denote  the  other  flow.  Assume  that  the 
normalized  allocations  of  resource  r  are  Za  and  for  the  two 
flows.  It  can  be  verified  that  the  normalized  allocations  of  the 
other  resource  are  3zq.  and  zy/i,  independent  of  whether  the 
resource  r  is  the  processing  resource  P  or  the  bandwidth  re¬ 
source  L.  Since  resource  r  is  the  most  congested  resource  as 
measured  by  the  sum  of  the  normalized  allocations,  we  have, 

3Zci  +  ZyjS  Zct  Zy 

which  leads  to  3Za  <  zy.  Since  both  flow  have  a  high  demand, 
under  the  max-min  notion,  this  condition  cannot  lead  to  a  fair 
allocation  except  for  the  trivial  case  where  Za  =  zy  =  0.  Thus, 
it  may  not  be  possible  to  achieve  a  fair  allocation  of  the  most 
congested  resource  as  measured  by  the  sum  of  the  normalized 
allocations  of  the  resource. 

Based  on  the  discussions  above,  we  claim  that  in  a  network 
where  no  explicit  preference  of  one  resource  over  another  ex¬ 
ists  (i.e.,  each  resource  is  essential),  fairness  should  not  be  de¬ 
fined  based  only  on  a  single  resource,  no  matter  how  this  single 
resource  is  determined,  and  whether  it  is  determined  before  al¬ 
location  (i.e.,  based  on  demand)  or  after  allocation  (i.e.,  based 
on  allocation).  Instead,  the  fairness  in  such  a  system  should  be 
defined  with  overall  consideration  of  various  resources  involved 
in  the  system  and  the  relationships  between  the  demands  for  the 
various  resources. 

Given  this  observation,  one  may  propose  yet  another  scheme 
to  define  fairness:  the  sum  of  the  normalized  allocations  of  the 
resources  computed  for  each  flow  should  be  max-min  fair.  In 
the  previous  example  B,  this  leads  to  an  allocation  of  75  MHz 
of  processing  time  and  25  Mbps  of  bandwidth  for  flow  1,  and 
25  MHz  of  processing  time  and  75  Mbps  of  bandwidth  for  flow 
2.  In  this  case,  for  both  flows,  the  sum  of  the  normalized  allo¬ 
cations  of  the  two  resources  is  75/100  +  25/100  =  1.  While 
this  appears  to  be  a  reasonable  strategy  for  fair  allocation,  this 
scheme  of  fairness  cannot,  in  fact,  be  extended  to  other  situa¬ 
tions.  This  is  illustrated  by  example  C  described  below. 

Assume  that  both  flows  have  a  demand  of  100  MHz  for  re¬ 
source  P,  while  the  demands  for  resource  L  are  20  Mbps  and 
10  Mbps  for  flows  1  and  2,  respectively.  Note  that  in  this  exam¬ 
ple,  there  is  sufficient  link  bandwidth  available  for  the  demands 
of  both  flows,  i.e.,  the  flows  are  not  in  competition  for  resource 
L.  In  other  words,  the  system  regresses  into  an  allocation  of  a 
single  resource  P.  Applying  the  max-min  notion  of  fairness  on 
the  single  resource  P,  we  know  that  the  fair  allocation  would  be 
50  MHz  of  processing  time  for  each  flow,  leading  to  10  Mbps 
and  5  Mbps  of  bandwidth  for  flows  1  and  2,  respectively.  Thus, 
the  ideally  fair  allocation  leads  to  0.6  and  0.55  as  the  sum  of 
the  normalized  allocations.  Clearly,  if  we  were  to  not  to  be 
max-min  fair  in  the  sum  of  the  normalized  allocations  of  the 
resources  to  each  flow,  we  would  not  get  this  result.  This  illus¬ 
trates  that  the  strategy  of  achieving  max-min  fair  distribution  in 
the  sum  of  the  normalized  allocations  fails  to  serve  as  the  basis 
to  define  fairness  in  the  allocation  of  multiple  resources. 

The  fair  allocation  strategies  in  the  three  examples  do  have 
one  property  in  common:  the  largest  normalized  allocations  of 
the  flows  are  distributed  in  a  max-min  fair  manner  among  the 
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flows.  In  our  case  with  equal  weights  for  the  flows,  the  largest 
normalized  allocations  are  equal  for  the  two  flows.  In  the  first 
two  examples  in  Table  I,  resource  P  is  the  prime  resource  for 
flow  1,  while  the  prime  resource  for  flow  2  is  resource  L.  In 
both  examples,  the  largest  normalized  allocation  equals  0.9  for 
each  flow.  In  the  third  example,  the  processor  P  is  the  prime 
resource  for  both  flows,  and  this  time  the  largest  normalized 
allocation  is  0.5  for  both  flows. 

The  observations  from  the  above  examples  lead  to  the  sig¬ 
nificance  of  incorporating  the  largest  normalized  allocation  for 
each  flow  into  a  strategy  for  extending  a  notion  of  fairness  to 
the  allocation  of  multiple  resources.  In  our  examples,  the  fair 
allocation  policy  is  to  simply  equalize  the  largest  normalized  al¬ 
locations  for  different  flows.  In  the  general  situation,  different 
notions  of  fairness  may  be  used  and  flows  may  have  different 
weights,  different  largest  normalized  demands,  and  very  differ¬ 
ent  utility  functions.  This  leads  to  the  following  Principle  of 
Fair  Essential  Resource  Allocation. 

Principle  of  Fair  Essential  Resource  Allocation  (FERA):  In 
a  system  with  multiple  related  and  essential  resources,  an  allo¬ 
cation  policy  q  is  said  to  be  fair  as  per  the  notion  of  fairness  T, 
if  and  only  if,  the  largest  normalized  allocations  are  distributed 
fairly,  as  per  the  notion  of  fairness  T,  with  respect  to  the  largest 
normalized  demands.  In  other  words,  allocation  policy  q  is  fair 
as  per  T  if  and  only  if, 

where  C  is  some  constraint  imposed  on  the  system. 

C.  Fair  Work-Conserving  Allocation  Policy 

Recall  that  we  make  no  assumption  on  whether  or  not  the 
allocation  policy  is  work-conserving.  Therefore,  under  differ¬ 
ent  constraints,  a  single  system  can  have  more  than  one  fair 
allocation  policy  as  per  the  same  notion  of  fairness.  Given  a 
constraint,  however,  there  exists  a  unique  work-conserving  fair 
allocation  policy  in  most  situations,  as  will  be  proved  in  this 
section. 

First,  we  formally  define  a  work-conserving  policy  in  the  al¬ 
location  of  multiple  resources.  Recall  that  in  the  allocation  of  a 
single  resource,  an  allocation  policy  is  work-conserving  if  and 
only  if  one  of  the  following  two  situations  occurs. 

1)  The  demands  of  all  flows  are  satisfied. 

2)  The  shared  resource  is  completely  allocated. 

In  other  words,  no  more  of  the  resource  can  be  further  allocated 
to  the  flows.  The  same  idea  can  be  applied  to  the  allocation  of 
multiple  resources,  except  that  now  it  is  possible  that  only  one 
resource  is  fully  utilized. 

Definition  5:  In  the  allocation  of  multiple  resources,  an  al¬ 
location  policy  is  said  to  be  work-conserving,  if  and  only  if, 
upon  completion  of  the  allocation,  no  more  of  any  resource  can 
be  further  allocated  to  a  flow  without  also  reducing  the  amount 
of  some  resource  allocated  to  another  flow. 

Next  we  introduce  two  general  classes  of  fairness  notions 
which  describe  the  conditions  under  which  the  uniqueness  of 
the  fair  work-conserving  allocation  policy  will  hold. 

Definition  6:  A  notion  of  fairness  T  is  said  to  be  uniquely 
deterministic,  if  and  only  if,  given  the  constraint  C,  the  nor¬ 
malized  demand  vector  [Di],  and  the  weight  vector  [rui],  one 


can  uniquely  determine  the  normalized  allocation  vector  [Ai] 
as  given  in  (1). 

Definition  7:  A  notion  of  fairness  F  is  said  to  be  non¬ 
decreasing,  if  and  only  if,  given  the  normalized  demand  vector 
\Di]  and  the  weight  vector  [rui],  the  normalized  allocation  [Ai] 
is  such  that,  for  any  two  different  constraints  Ci  and  C2,  one  of 
the  following  holds  true: 

F{Ci,[D,],[w,])  -i  F  {C2,[D,],[w,]) 
.F(C'2,[A],M)  .F(Ci,[A],K]). 

Here  is  a  relational  operator  between  two  vectors  of  identical 
dimensions,  and  [ui]  -<  [v^]  implies  <  Vi.  This  definition 
of  non-decreasing  fairness  notion  can  be  also  expressed  as  fol¬ 
lows:  when  allocating  a  single  resource  under  a  non-decreasing 
fairness  notion,  no  flow  will  get  a  lesser  amount  of  the  resource 
if  the  total  amount  of  the  shared  resource  increases. 

These  classes  of  fairness  notions  are  actually  very  broad;  it 
may  be  readily  verified  that  many  popular  notions  of  fairness 
are  both  non-decreasing  and  uniquely  deterministic.  These  in¬ 
clude  max-min  fairness  [3-5],  proportional  fairness  [6],  and 
utility  max-min  fairness  [7]  if  the  utility  functions  are  non¬ 
decreasing. 

Theorem  1:  If  the  applied  notion  of  fairness  is  both  non¬ 
decreasing  and  uniquely  deterministic,  there  exists  a  unique  fair 
work-conserving  allocation  policy  that  satisfies  the  FERA  prin¬ 
ciple  as  stated  in  Section  III-B. 

Proof:  The  reader  is  referred  to  Appendix  I.  ■ 

IV.  Fair  Joint  Allocation  of  Processing  and 
Bandwidth  Resources 

In  this  section,  we  apply  the  framework  established  in  the 
previous  section  into  an  important  context  of  special  interest: 
the  fair  joint  allocation  of  a  shared  processor  P  and  a  shared 
link  L  under  the  max-min  notion  of  fairness  and  linear  utility 
functions. 

A.  System  Model 

In  this  system  model,  a  set  of  N  flows  share  a  processor  P 
and  a  link  L,  as  shown  in  Fig.  2.  Packets  from  each  flow  are 
processed  by  processor  P  first  and  then  transmitted  onto  the 
output  link  L.  Denote  by  Rl  the  peak  bandwidth  rate  of  link 
L  and  hy  Rp  the  peak  processing  rate  of  processor  P.  Pack¬ 
ets  of  each  flow  await  processing  by  the  processor  in  an  input 
buffer  of  infinite  capacity,  and  then  upon  completion  of  the  pro¬ 
cessing,  await  transmission  on  the  output  link  in  another  buffer 
of  infinite  capacity.  The  joint  allocation  of  the  processing  and 
bandwidth  resources  is  accomplished  by  the  scheduler  which 
acts  on  the  packets  in  the  input  buffers  and  appropriately  orders 
them  for  processing  by  the  processor.  No  scheduling  action 
takes  place  after  the  processing;  processed  packets  are  received 
in  the  buffer  between  the  processor  and  the  link,  and  are  trans¬ 
mitted  in  a  first-come-first-served  fashion. 

Denote  by  Wi  the  weight  of  flow  i,  1  <  i  <  N,  indicating  the 
flow’s  relative  rightful  share  of  the  resources. 
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Flow  1 
Flow  2 
Flow  N 


Scheduler 


Fig.  2.  The  system  model  with  a  shared  processor  P  and  a  shared  link  L. 


B.  Fluid-flow  Processor  and  Link  Sharing 

Denote  by  S  the  system  illustrated  in  Fig.  2.  We  first  con¬ 
sider  fluid-flow  traffic  through  system  S,  and  describe  an  ide¬ 
ally  fair  allocation  strategy  called  the  Fluid-flow  Processor  and 
Link  Sharing  (FPLS)  algorithm.  FPLS  is  intended  to  serve  the 
same  purpose  for  system  S  as  that  served  by  Generalized  Pro¬ 
cessor  Sharing  (GPS)  for  a  system  with  just  a  single  shared  link 
or  a  single  shared  processor  [4,5]. 

In  GPS,  it  is  assumed  that  traffic  from  each  flow  can  be  di¬ 
vided  into  infinitesimally  small  chunks,  and  each  chunk  has  its 
demand  for  access  to  the  link  L  depending  on  the  size  of  the 
chunk.  The  GPS  scheduler  visits  each  active  flow’s  queue  in  a 
round-robin  fashion,  and  serves  an  infinitesimally  small  amount 
of  data  from  each  queue,  in  such  a  way  that  during  any  infinites¬ 
imal  interval  of  time,  it  can  visit  each  queue  at  least  once.  In  our 
study,  this  assumption  is  still  valid,  and  we  further  assume  that 
each  infinitesimal  chunk  also  has  its  demand  for  the  processing 
time  on  the  shared  processor  P. 

At  each  time  instant  r,  the  prime  resource  for  each  flow,  ac¬ 
cording  to  Definition  4,  can  be  determined  based  on  its  instan¬ 
taneous  demands  for  processing  time  and  bandwidth.  In  addi¬ 
tion,  we  assume  that  during  each  infinitesimal  interval  of  time, 
[t,  t  4-  At),  the  prime  resource  for  each  flow  does  not  change. 

Note  that  in  GPS,  it  is  guaranteed  that  during  each  infinites¬ 
imal  interval  of  time,  the  chunks  of  each  flow  are  scheduled  in 
such  a  way  that,  for  each  flow,  the  total  demand  for  bandwidth 
corresponding  to  the  chunks  of  the  flow  scheduled  in  this  period 
is  proportional  to  the  weight  of  the  flow.  Extending  GPS  to  our 
case  leads  to  the  following:  under  the  ideally  fair  allocation  pol¬ 
icy  for  system  S,  it  is  guaranteed  that,  during  each  infinitesimal 
interval  of  time,  the  chunks  of  each  flow  are  scheduled  in  such 
a  way  that,  for  each  flow,  the  total  normalized  demand  for  its 
prime  resource  corresponding  to  the  chunks  of  the  flow  sched¬ 
uled  in  this  period  is  proportional  to  the  weight  of  the  flow.  We 
refer  to  this  as  Fluid-flow  Processor  and  Link  Sharing  (FPLS). 
It  can  be  readily  verified  that  the  FPLS  strategy  meets  the  FERA 
principle  described  in  Section  III-B. 


C.  Packet-by-packet  Processor  and  Link  Sharing 

It  is  apparent  that  EPLS  is  an  ideally  fair  but  unimple- 
mentable  policy,  in  the  same  sense  as  GPS.  In  reality,  network 
traffic  is  always  packetized,  and  therefore,  we  next  present  a 
practical  approximation  of  EPLS,  called  Packet-by-packet  Pro¬ 
cessor  and  Link  Sharing  (PPLS).  The  pseudo-code  of  PPLS  is 
shown  in  Pig.  3. 

The  PPLS  algorithm  extends  one  of  the  most  practical  and 
simple  scheduling  strategies.  Deficit  Round  Robin  (DRR)  [10], 
used  in  the  allocation  of  bandwidth  on  a  link.  Por  each  flow, 
the  DRR  algorithm  maintains  a  deflcit  counter  (DC),  which  is 


1  Initialize: 

2  FlowList  <—  NULL; 

3  Enqueue:  /*  Invoked  whenever  a  packet  arrives  */ 

4  p  ^  ArrivingPacket; 

5  %  ^  Flow(p)-,  /*  Flow  of  packet  p  */ 

6  if  (ExistsInFlowList(i)  =  FALSE)  then 

7  Append  flow  i  to  FlowList, 

8  PDCi  ^  0; 

9  LDCi  ^  0; 

10  end  if; 

1 1  Dequeue:  /*  Always  running  */ 

12  while  (TRUE)  do 

13  if  (FtowLwf  7^  NULL)  then 

14  i  ^  HeadOf FlowList, 

15  Remove  i  from  F/owLi.sf; 

16  PDCi  ^  PDCi  +  PQt, 

17  LDC^  ^  LDCi  +  LQi-, 

18  if  (PDCi  >  mavPDGi)  then 

19  PDCi  ^  maxPDCi', 

20  end  if; 

21  if  {LDCi  >  maxLDCi)  then 

22  LDCi  ^  maxLDCi', 

23  end  if; 

24  while  (QueueIsEmpty{i)  =  EALSE)  do 

25  p  ^  HeadOfLinePacketInQueue{iy, 

26  if  (Sizeip)  >  LDCi  OR 

27  ProcessingCost{p)  >  PDCi)  then 

28  break;  /*  escape  from  the  inner  while  loop  */ 

29  end  if; 

30  PDCi  <—  PDCi—  ProcessingCostip)', 

31  LDCi  ^  LDCi  —  Size{p); 

32  Schedule  p; 

33  end  while; 

34  if  {QueuelsEmptyii)  =  EALSE)  then 

35  Append  queue  i  to  FlowList, 

36  end  if; 

37  end  if; 

38  end  while; 


Fig.  3.  Pseudo-code  of  the  Packet-by-packet  Processor  and  Link  Sharing 
(PPLS)  algorithm. 


incremented  in  each  round  by  a  predetermined  quantity,  quan¬ 
tum.  When  the  scheduler  visits  one  flow,  it  transmits  the  pack¬ 
ets  from  this  flow  with  a  total  length  no  more  than  the  deficit 
counter  associated  with  this  flow.  Upon  the  completion  of  a 
flow’s  service  opportunity,  its  deficit  counter  is  decremented  by 
the  total  size  of  its  packets  scheduled  in  the  round.  It  has  been 
shown  in  [10]  that,  if  the  quantum  of  each  flow  is  proportional 
to  its  weight,  the  relative  fairness  measure  as  defined  in  [9]  can 
be  bounded. 

The  PPLS  algorithm  approximates  the  ideal  EPLS  in  a  very 
similar  fashion  as  DRR  achieves  an  approximation  of  GPS.  The 
PPLS  scheduler  maintains  a  linked  list  of  the  backlogged  flows, 
FlowList.  When  the  scheduler  is  initialized,  FlowList  is  set  to 
an  empty  list  (line  2).  Lor  each  flow,  two  variables,  instead  of 
one  as  in  DRR,  are  maintained  in  the  PPLS  algorithm:  a  proces¬ 
sor  deflcit  counter  (PDC)  and  a  link  deflcit  counter  (LDC).  The 
link  deficit  counter  is  exactly  the  same  as  the  deficit  counter  in 
DRR,  which  represents  the  deviation  of  the  bandwidth  received 
by  the  flow  from  its  ideally  fair  share.  The  processor  deficit 


counter,  on  the  other  hand,  represents  the  deviation  of  the  pro¬ 
cessing  time  allocated  to  the  flow  from  its  ideally  fair  share. 
Thus,  each  flow  in  PPLS  is  assigned  two  quantum  values,  a 
processor  quantum  (PQ)  and  a  link  quantum  (LQ). 

When  a  new  packet  arrives,  the  Enqueue  procedure  is  in¬ 
voked  (lines  3-10).  If  this  packet  comes  from  a  new  flow,  the 
Enqueue  procedure  appends  this  flow  to  the  end  of  the  FlowList 
(line  7)  and  initializes  both  of  its  deficit  counters  to  0  (lines  8- 

9). 

The  Dequeue  procedure  (lines  1 1-38)  functions  as  follows.  It 
serves  all  flows  in  the  FlowList  in  a  round-robin  fashion.  When 
the  scheduler  visits  flow  i,  it  first  increments  each  of  the  two 
deficit  counters  of  this  flow  by  the  value  of  the  corresponding 
quantum  (lines  16-17).  It  then  verifies  whether  or  not  these  two 
deficit  counters  exceed  their  upper  bounds  respectively,  and  if 
they  do,  it  resets  them  to  the  maximum  possible  values  (lines 
18-23).  The  rationale  behind  this  bounding  process  will  be  dis¬ 
cussed  later  in  detail.  After  the  deficit  counters  of  flow  i  are 
updated,  a  sequence  of  packets  from  flow  i  are  scheduled  as 
long  as  the  total  length  of  these  packets  is  smaller  than  the  link 
deficit  counter,  and  the  total  processing  cost  is  smaller  than  the 
processing  deficit  counter,  as  in  the  while  loop  in  lines  24—33. 
In  the  meantime,  when  a  packet  is  scheduled,  both  deficit  coun¬ 
ters  are  decremented  by  the  corresponding  cost  of  this  packet 
(lines  30-31).  Finally,  when  the  scheduler  finishes  serving  a 
flow  and  the  flow  still  remains  backlogged,  the  scheduler  places 
the  flow  back  at  the  end  of  the  FlowList  (lines  34-36). 

Recall  that  in  DRR,  for  each  flow,  the  quantum  is  set  to  be 
proportional  to  its  weight,  therefore,  each  flow  receives  in  each 
round,  on  average,  a  service  of  total  amount  proportional  to  its 
weight.  In  this  report,  the  sum  of  a  certain  quantity  over  all 
flows  is  denoted  by  dropping  the  subscript  for  the  flow  in  the 
notation.  For  example,  w  is  the  sum  of  the  weights  for  all  flows, 
i.e.,  w  =  Wi.  Therefore,  in  DRR,  we  have 

Q^  Q 

—  =  — ,Vz. 

Wi  w 


Similarly,  in  PPLS,  the  quantum  values  of  each  flow  are  also 
proportional  to  its  weight,  i.e.,  Vz, 


PQ, 

PQ 

(2) 

=  - 

Wi 

W 

LQ, 

LQ 

(3) 

Wi 

w 

Thus  the  amount  of  the  shared  resources  each  flow  is  entitled 
to  utilize  in  each  round  is  guaranteed  to  be,  on  average,  propor¬ 
tional  to  its  weight.  In  addition,  the  ratio  of  the  sum  of  process¬ 
ing  quanta  for  all  flows,  PQ,  to  the  sum  of  link  quanta  for  all 
flows,  LQ,  should  also  be  equal  to  the  ratio  of  the  total  amount 
of  processing  resource  to  the  total  amount  of  link  resource  in 
each  round,  i.e.. 


PQ  Rp 
LQ~  Rl 


(4) 


From  (2),  (3)  and  (4),  it  is  apparent  that. 


PQi  _  Rp 

LQi  Rp 


(5) 


In  other  words,  for  each  flow,  the  quantum  value  corresponding 
to  a  resource  is  proportional  to  the  total  amount  of  that  resource. 

Note  that  in  PPLS,  it  is  possible  that  the  prime  resource  for 
flow  i  remains  the  same  for  a  long  period,  and  therefore,  with¬ 
out  the  bounding  procedure  in  lines  18-23,  the  deficit  counter 
for  the  non-prime  resource  would  reach  a  large  value.  For  ex¬ 
ample,  consider  a  case  in  which  the  prime  resource  for  flow 
i  has  been  the  processing  resource  P  for  a  long  time  and,  as 
a  result,  the  link  deficit  counter  LDCi  is  very  large.  Assume 
that  at  this  point,  the  prime  resource  for  flow  i  switches  to  the 
link  resource  L  and,  in  addition,  flow  i  now  consumes  almost 
no  processing  resource.  In  such  a  situation,  flow  i  will  be  able 
to  have  a  long  sequence  of  packets  scheduled  because  of  its 
large  link  deficit  counter  LDCi.  This  would  significantly  de¬ 
grade  the  short-term  fairness  of  the  PPLS  scheduler.  For  this 
reason,  we  choose  to  set  a  maximum  threshold  on  the  deficit 
counter  for  each  resource,  in  case  any  specific  resource  has  not 
been  fully  utilized  for  a  long  time.  In  cases  where  short-term 
fairness  is  not  important,  these  thresholds  may  simply  be  set 
to  infinity.  A  similar  rationale  may  also  be  found  in  the  con¬ 
text  of  fair  scheduling  in  wireless  networks  where  a  maximum 
lag  is  applied  when  a  flow  has  not  fully  utilized  its  share  of  the 
bandwidth  [18]. 

It  can  be  readily  verified  that  if  the  processor  resource  P  is 
sufficient  for  all  flows,  i.e.,  the  processor  resource  P  never  be¬ 
comes  the  prime  resource  for  any  flow,  the  PPLS  strategy  re¬ 
gresses  into  the  DRR  policy.  It  can  also  be  readily  verified  that, 
like  DRR,  the  per-packet  computing  complexity  of  the  PPLS 
algorithm  is  0(1),  under  the  condition  that  for  each  flow  i, 
LQi  >  Mp  and  PQi  >  Mp  where  Mp  and  Mp  are  the  maxi¬ 
mum  packet  size  and  the  maximum  packet  processing  cost,  re¬ 
spectively.  The  proof  of  this  work  complexity  is  similar  to  that 
for  DRR  [10]. 


D.  Fairness  Analysis  of  PPLS 

Our  fairness  analysis  of  PPLS  is  an  extension  of  that 
in  [10].  Recall  that  in  bandwidth  sharing,  the  fairness 
measure,  FM(fi,f2),  is  defined  as  the  maximum  value  of 
Senti  {ti  ,t2)jwi  —  Sentj  (fi ,  ^2) /wj  amongst  all  pairs  of  flows 
{i,j)  backlogged  in  the  interval  [^1,^2),  where  Senti{ti,t2) 
is  the  total  length  of  the  packets  transmitted  by  the  scheduler 
from  flow  i  during  interval  [^1,^2),  i-C-,  the  cumulative  amount 
of  bandwidth  resource  allocated  to  flow  i  during  this  interval. 
In  addition,  the  fairness  bound  FB  is  defined  as  the  maximum 
value  of  FM(fi,  ^2)  for  all  possible  intervals  [ti,t2)  [10].  Note 
that  the  dimension  of  Senti{ti,t2)  is  in  units  of  packet  length, 
i.e.,  bytes.  To  extend  this  concept  to  our  situation  of  multiple 
heterogeneous  resources,  we  need  to  normalize  it  by  the  peak 
resource  consumption  rate. 

The  normalized  cumulative  processor  allocation  of  flow  i 
during  time  interval  [^1,^2)^  denoted  by  CPA"(fi,f2),  is  de¬ 
fined  as  the  total  amount  of  the  processing  resource  allocated 
to  flow  i  during  interval  [ti,t2),  normalized  by  the  peak  pro¬ 
cessing  rate  Rp.  The  normalized  cumulative  link  allocation  of 
flow  i  during  time  interval  [ti,t2),  denoted  by  CLA"(fi,f2), 
is  similarly  defined,  except  that  the  resource  associated  is  the 
bandwidth. 
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Note  that  both  the  normalized  cumulative  link  allocation  and 
the  normalized  cumulative  processor  allocation  are  in  units  of 
time.  Therefore,  we  are  able  to  proceed  to  define  the  normal¬ 
ized  cumulative  resource  allocation  of  flow  i  during  time  in¬ 
terval  [ti,t2),  denoted  by  CRA"(fi,  ^2),  as  the  larger  value  be¬ 
tween  the  normalized  cumulative  processor  and  link  allocations 
of  flow  i  during  [ti  ^t^).  In  other  words, 


CRA”  (f  1 ,  f2)  =  max{CPA”  (f  1 ,  fa ) ,  CLA”  (f  1 ,  fa) }  • 


Now  we  can  extend  the  definition  of  the  fairness  measure  as 
follows: 

Definition  8:  The  normalized  fairness  measure  over  time 
interval  [fi,  fa),  FM”(fi,  fa),  is  defined  as  the  maximum  value, 
amongst  all  pairs  of  flows  {i,j)  that  are  backlogged  during  time 
interval  [fijfa),  of  the  normalized  cumulative  resource  alloca¬ 
tion  CRA"(fi,f2).  That  is, 

CRA" (f  1,^2)  _  CRA"(fi,f2) 

Wi  Wj 


FM”(fi,  fa)  =  max 
V(*j) 


TABLE  II 

The  ratio  of  the  processing  resource  to  the  link  resource 

REQUIRED  BY  EACH  ELOW  IN  OUR  STUDY  USING  SYNTHETIC  TRAEFIC. 


Flow  ID 

P/ L  Ratio  (in  cycles/byte) 

1 

1 

2 

2 

3 

3 

4 

4 

5 

1 

6 

1/2 

7 

1/3 

8 

1/4 

In  the  following,  we  describe  two  sets  of  simulation  exper¬ 
iments.  In  the  first  set  of  experiments,  a  synthetic  traffic  se¬ 
quence  is  used,  while  the  second  set  uses  real  gateway  traffic 
traces. 


The  normalized  fairness  bound  FB”  is  defined  as  the  maximum 
value  of  the  normalized  fairness  measure  FM”(fi,  fa)  over  all 
possible  intervals  [fijfa). 

Analogous  to  the  case  of  a  single  shared  resource,  if  a 
scheduling  algorithm  for  the  joint  allocation  of  processing  and 
bandwidth  resources  leads  to  a  finite  normalized  fairness  bound, 
one  can  conclude  that  this  algorithm  approximates  the  ideally 
fair  allocation  and  achieves  long-term  fairness.  The  following 
theorem  states  this  about  the  PPLS  algorithm. 

Theorem  2:  The  normalized  fairness  bound  of  PPLS  is  a 
finite  constant. 

Proof:  The  reader  is  referred  to  the  Appendix  II.  ■ 

V.  Simulation  Results  and  Analysis 

Our  simulation  model  consists  of  8  flows  with  equal  weights 
sharing  a  processor  P  and  a  link  L,  as  shown  in  Fig.  1(a).  Five 
different  scheduling  policies  including  the  PPLS  algorithm  are 
implemented. 

•  FCFS  (First-Come  First-Served):  A  simple  FCFS  scheme 
is  used.  The  scheduling  order  is  only  determined  by  the 
packet  timestamps. 

•  PPLS:  When  the  PPLS  algorithm  is  implemented,  a  FCFS 
strategy  is  used  on  the  buffer  between  the  processor  P  and 
the  link  L,  since  the  order  of  the  packets  has  already  been 
determined  by  the  PPLS  algorithm. 

•  LDRR  (Link  Deficit  Round  Robin):  A  DRR  algorithm  in 
the  allocation  of  only  the  link  bandwidth  is  implemented 
(i.e.,  the  original  DRR). 

•  PDRR  (Processor  Deficit  Round  Robin):  A  DRR  algo¬ 
rithm  in  the  allocation  of  only  the  processing  resource  is 
implemented. 

•  DDRR  (Double  Deficit  Round  Robin):  Two  DRR  sched¬ 
ulers  are  used.  PDRR  is  used  before  the  processor  P  and 
LDRR  is  used  before  the  link  L.  Note  that  this  is  the  only 
scheme  in  which  a  scheduler  is  implemented  between  the 
processor  and  the  link. 


A.  Synthetic  Traffic 

In  our  first  study,  we  use  synthetic  traffic  to  test  the  fairness 
properties  of  the  PPLS  algorithm  under  some  extreme  situa¬ 
tions.  For  each  flow,  the  ratio  of  the  amount  of  the  processing 
resource  required  to  the  amount  of  the  bandwidth  resource  re¬ 
quired  is  a  fixed  value.  Note  that  in  the  definition  of  the  normal¬ 
ized  fairness  measure,  FM"(fi,  fa),  if  both  the  Rp  and  Rp  are 
multiplied  by  the  same  value,  the  normalized  fairness  measure 
will  also  be  multiplied  by  this  value.  In  other  words,  the  fact 
of  whether  or  not  the  normalized  fairness  measure  is  bounded 
does  not  change  except  that  the  bound  itself  may  vary.  There¬ 
fore,  for  better  illustration  and  easier  comparison,  we  normalize 
the  resource  amounts  such  that  the  average  number  of  proces¬ 
sor  cycles  needed  per  packet  is  numerically  equal  to  the  average 
number  of  bytes  per  packet.  Table  II  shows  the  ratios  used  in 
this  study.  Note  that  flows  1  and  5  have  equal  normalized  de¬ 
mands  for  both  resources.  The  prime  resource  for  flows  2,  3 
and  4  is  the  processor,  and  that  for  flows  6,  7  and  8  is  the  link. 
For  flows  1  to  4,  the  sizes  of  packets  generated  is  uniformly  dis¬ 
tributed  between  1  and  1,600  bytes,  while  for  flows  5  to  8,  the 
processing  cost  is  uniformly  distributed  between  1  and  1,600 
cycles.  Therefore,  the  maximum  packet  size  is  6,400  bytes  and 
the  maximum  processing  cost  is  6,400  cycles.  These  maximum 
values  are  the  quantum  values  assigned  to  each  flow. 

Fig.  4  shows  the  normalized  cumulative  resource  allocations, 
CRA"(0,  r),  for  all  flows  i  after  a  long  run  in  the  simulations. 
In  this  plot,  the  fairer  an  allocation  policy,  the  closer  its  corre¬ 
sponding  curve  to  a  straight  horizontal  line.  It  is  apparent  from 
this  figure  that  the  PPLS  scheduling  policy  does  achieve  good 
fairness.  Note  that,  as  expected,  the  FCFS  scheme  is  the  worst 
among  all  in  terms  of  fairness.  Regarding  LDRR  and  PDRR, 
each  can  achieve  fair  distribution  of  the  normalized  cumulative 
allocation  with  respect  to  a  certain  resource,  but  not  the  overall 
normalized  cumulative  resource  allocation.  Take  LDRR  as  an 
example.  It  achieves  fair  distribution  of  the  normalized  cumula¬ 
tive  link  allocation  for  all  flows.  Therefore,  flows  with  the  pro¬ 
cessor  as  the  prime  resource,  namely  flows  2  to  4  in  this  case. 
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Fig.  4.  The  simulation  results  using  synthetic  traffic.  In  this  plot,  the  fairer 
an  allocation  policy,  the  closer  its  corresponding  curve  to  a  straight  horizontal 
line. 

receive  a  larger  amount  of  normalized  cumulative  processor  al¬ 
location  as  compared  to  other  flows.  This  indicates  that  LDRR 
fails  to  achieve  overall  fairness  among  flows.  PDRR  functions 
exactly  in  the  opposite  way:  it  fairly  distributes  the  normalized 
cumulative  processor  allocation  among  all  flows,  but  flows  with 
the  link  as  the  prime  resource  (flows  5  to  8)  receive  a  large  nor¬ 
malized  cumulative  link  allocation.  Thus,  PDRR  also  fails  to 
achieve  overall  fairness. 

One  interesting  observation  is  regarding  the  DDRR  scheme. 
Intuitively,  one  may  expect  DDRR  to  serve  as  a  fair  scheduler 
in  the  allocation  of  processing  and  bandwidth  resources,  since 
it  has  two  schedulers,  one  fair  with  respect  to  the  processor  and 
the  other  fair  with  respect  to  the  link.  However,  Fig.  4  shows 
that  this  is  not  the  case.  This  is  because  the  DDRR  scheme  im¬ 
plements  the  two  fair  schedulers  in  different  stages.  Note  that 
the  PDRR  scheduler  before  the  processor  P  is  responsible  for 
fairly  allocating  the  normalized  cumulative  processor  allocation 
to  all  flows.  That  means,  at  this  point,  more  packets  (in  bytes) 
from  those  flows  with  the  link  as  the  prime  resource  (flows  6  to 
8)  will  be  scheduled  from  the  processor  P.  On  the  other  hand, 
those  flows  with  the  processor  as  the  prime  resource  (flows  2 
to  4)  will  not  have  enough  packets  remaining  backlogged  in 
the  buffer  before  the  link  L.  The  LDRR  scheduler,  therefore, 
finds  flows  2  to  4  to  be  relatively  idle  and  ends  up  transmitting 
more  packets  from  flows  6  to  8,  thus  causing  a  higher  normal¬ 
ized  cumulative  resource  allocation  for  flows  6  to  8.  In  fact, 
the  DDRR  scheme  allocates  resources  fairly  to  all  flows  with 
the  same  prime  resource,  but  favors  the  flows  with  the  “final” 
resource  as  the  prime  resource. 

B.  Gateway  Traffic  Traces 

In  this  study,  we  use  real  traffic  recorded  at  an  Internet  gate¬ 
way  as  the  input  traffic  [14, 19].^  The  traffic  traces  include  the 
processing  delay  (in  milliseconds)  for  each  packet,  along  with 

^Global  Positioning  System  technology  was  used  to  precisely  record  the 
timestamp  of  each  packet  at  each  node.  In  the  trace  data,  filtered  IP  headers 
were  examined  to  track  the  same  packet  at  different  nodes.  The  difference  be¬ 
tween  the  timestamps  of  the  same  packet  at  adjacent  nodes  was  computed  as 
the  delay.  The  link  speed  connecting  these  nodes  was  taken  into  consideration 
so  that  the  transmission  delay  of  each  packet  was  removed  from  the  recorded 
delay.  Note  that  this  delay  was  still  the  sum  of  the  processing  delay  and  the 
queueing  delay.  However,  it  was  noticed  that  for  traffic  in  one  of  the  two  direc- 
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Fig.  5.  The  simulation  results  using  gateway  traffic  traces,  (a)  The  case  when 
maxPDCi  and  maxLDCi  are  large  enough  for  all  i  such  that  lines  19  and  22 
in  Fig.  3  are  never  executed,  (b)  The  effect  of  maxPDCi  and  maxLDCi  in  the 
PPLS  algorithm.  Here,  for  all  i,  we  assume  maxPDCi  =  maxLDC i  = 
maxDC  and  PQi  =  LQi  =  Q-  Again,  in  these  plots,  the  fairer  an  allocation 
policy,  the  closer  its  corresponding  curve  to  a  straight  horizontal  line. 

the  packet  size  (in  bytes).  For  our  experiments,  we  assume  a 
fixed  processing  rate,  and  correspondingly  convert  the  process¬ 
ing  delay  of  each  packet  into  processor  cycles.  Again,  we  con¬ 
vert  the  processing  delay  of  each  packet  in  such  a  way  that  the 
average  processing  delay  per  packet  (in  units  of  cycles)  is  nu¬ 
merically  equal  to  the  average  size  per  packet  (in  units  of  bytes). 
To  achieve  a  better  comparison  with  the  previous  study  using 
synthetic  traffic,  the  flows  have  been  ordered  in  such  way  that 
the  overall  prime  resource  for  flows  1  to  4  is  the  processor,  and 
that  for  flows  5  to  8  is  the  link. 

Fig.  5(a)  illustrates  the  normalized  cumulative  resource  allo¬ 
cation  for  the  five  scheduling  policies  in  this  experiment.  Again, 
the  PPLS  algorithm  performs  very  well  in  terms  of  fairness.  It 
is  observed  that  all  other  conclusions  drawn  from  the  synthetic 
study  are  still  valid. 

Note  that  in  this  study,  the  DDRR  scheme  performs  closer  to 
the  PPLS  algorithm  than  in  the  previous  study.  This  can  be  at¬ 
tributed  to  the  fact  that  in  real  traces,  the  demands  of  each  flow 
for  the  processing  and  bandwidth  resources  are  more  balanced 
than  those  in  the  synthetic  traffic.  However,  this  would  not  be 
valid  if  a  flow  has  a  dominant  demand  for  one  resource  in  com¬ 
parison  to  another,  as  might  happen  during  a  DoS  attack.  In 

tions,  the  queue  occupancy  was  never  above  1  packet,  and  this  eliminates  the 
queueing  delay  and  validates  the  use  of  this  delay  as  the  pure  processing  delay. 
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addition,  the  PPLS  algorithm  only  needs  one  scheduler  in  real 
implementation  while  the  DDRR  needs  two. 

C.  Effect  of  Maximum  Deficit  Counter  Values 

Note  that  in  the  synthetic  study,  no  flow  changes  its  prime  re¬ 
source  during  the  experiment.  Therefore,  the  setting  of  the  max¬ 
imum  values  of  the  deficit  counters  (maxPDCi  and  maxEDCf) 
in  the  PPLS  algorithm  has  no  effect  on  the  outcome  of  the  sim¬ 
ulations.  Next,  using  the  real  gateway  traces,  we  focus  on  the 
effect  of  setting  the  maximum  values  of  deficit  counters  in  the 
PPLS  algorithm.  This  is  shown  in  Fig.  5(b),  where  we  assume 
that,  for  all  i,  maxPDCi  (in  units  of  processing  cycles)  is  nu¬ 
merically  equal  to  maxLDCi  (in  units  of  bytes). 

It  is  apparent  that  the  prime  resource  of  a  flow  changes  in  this 
study,  since  the  normalized  cumulative  resource  allocation  be¬ 
gins  to  show  differences  under  the  PPLS  algorithm.  However, 
it  should  be  noticed  that  the  normalized  cumulative  resource  al¬ 
locations  for  the  flows  are  still  reasonably  close  to  each  other, 
due  to  the  long-term  fairness  achieved  by  the  PPLS  algorithm. 

From  Fig.  5(b),  it  is  observed  that,  as  expected,  the  long-term 
fairness  among  normalized  cumulative  resource  allocations  de¬ 
grades  as  the  set  maximum  values  of  the  deficit  counters  de¬ 
crease.  For  example,  when  the  maximum  deficit  counters  are 
set  to  be  10  times  as  large  as  the  quantum  values,  the  normalized 
cumulative  resource  allocation  exhibits  a  10%  variation  from 
the  ideal. 

It  is  also  observed  from  Fig.  5(b)  that  flows  with  more  bal¬ 
anced  normalized  cumulative  allocations  between  the  two  re¬ 
sources  over  the  long  run,  such  as  flows  1,  2  and  5,  are  likely 
to  receive  less  normalized  cumulative  resource  allocation.  This 
may  be  attributed  to  the  fact  that  these  flows  are  more  likely  to 
temporarily  change  the  prime  resource,  and  therefore,  setting 
the  deficit  counter  for  the  current  non-prime  resource  to  the 
maximum  value  may  reduce  the  future  usage  of  this  resource 
when  it  later  becomes  the  prime  resource.  On  the  other  hand, 
the  unbalanced  flows  are  less  likely  to  temporarily  change  the 
prime  resource,  and  therefore,  the  effect  on  these  flows  of  set¬ 
ting  the  deficit  counter  for  the  non-prime  resource  to  the  max¬ 
imum  value  is  limited.  Similar  scenarios  may  also  be  found  in 
other  situations,  such  as  bandwidth  sharing.  For  example,  in 
DRR,  a  flow  that  frequently  changes  its  status  (between  being 
backlogged  and  not  being  backlogged)  will  be  sacrificed  in  the 
long  run,  since  each  time  it  becomes  non-backlogged  its  unused 
deficit  counter  is  reset  to  0,  thus  causing  it  to  lose  bandwidth 
share. 

Based  on  the  above  discussion,  by  setting  appropriate  max¬ 
imum  values  for  the  deficit  counters,  one  can  tune  the  trade¬ 
offs  between  long-term  and  short-term  fairness  achieved  by  the 
PPLS  algorithm.  This  is  similar  to  the  role  played  by  the  maxi¬ 
mum  lag  in  wireless  scheduling  [18]. 

VI.  Concluding  Remarks 

A.  Summary 

Research  in  fair  allocation  of  the  bandwidth  resource  has 
been  active  for  decades.  Traffic  flows,  however,  encounter 
multiple  resources  other  than  bandwidth,  including  processor, 
buffer,  and  power,  as  they  traverse  the  network.  Further,  the 


bandwidth  resource  is  not  always  the  sole  bottleneck  causing 
network  congestion.  In  this  report,  we  consider  a  set  of  shared 
resources  which  are  essential  and  related  such  as  processor, 
link  bandwidth,  and  power.  We  then  present  the  Principle 
of  Fair  Essential  Resource  Allocation,  or  the  FERA  principle, 
which  defines  the  fairness  in  the  joint  allocation  of  these  re¬ 
sources.  We  further  apply  the  FERA  principle  into  a  system 
consisting  of  a  shared  processor  and  a  shared  link,  and  propose 
a  practical  and  provably  fair  algorithm,  the  Packet-by-packet 
Processor  and  Link  Sharing  (PPLS),  for  the  joint  allocation  of 
the  processor  and  bandwidth  resources.  It  is  our  hope  that  this 
work  will  facilitate  future  research  on  achieving  provable  fair¬ 
ness  in  computer  networks. 

B.  A  Discussion  on  Implementation  of  PPLS 

In  this  report,  we  select  DRR  [10]  as  the  starting  point  in 
the  design  of  the  fair  allocation  policy  for  a  shared  processor 
and  a  shared  link,  because  of  its  relatively  simple  implementa¬ 
tion.  Other  fair  scheduling  algorithms  can  be  also  used,  such  as 
Weighted  Eair  Queueing  (WEQ)  [3],  Worst-case  Eair  Weighted 
Eair  Queueing  (WE^Q)  [11],  Surplus  Round  Robin  (SRR)  [20], 
and  Elastic  Round  Robin  (ERR)  [12]. 

Note  that  in  many  situations,  the  processing  cost  of  a  packet 
cannot  be  determined  before  it  is  actually  processed.  If  this 
is  the  case,  one  can  have  the  following  choices  to  modify  the 
PPLS  algorithm.  The  first  way  is  to  let  the  scheduler  predict 
the  processing  cost,  and  make  scheduling  decisions  based  on 
predicted  values.  In  the  second  choice,  the  scheduler  serves  the 
packet  first,  then  updates  the  deficit  counters  accordingly.  In 
this  way,  it  is  possible  that  after  serving  a  packet,  its  process¬ 
ing  deficit  counter  becomes  negative,  thus  breaking  the  fairness 
property  of  the  PPLS  algorithm.  Therefore,  the  scheduler  needs 
an  additional  counter  to  record  the  minimum  normalized  deficit 
counter  for  all  flows,  and  if  this  value  becomes  negative,  at  the 
beginning  of  next  round,  it  needs  to  add  a  proper  amount  to  the 
deficit  counter  of  each  flow  to  make  it  non-negative.  Note  that 
even  when  using  prediction  before  scheduling,  one  still  needs 
this  protection  from  a  negative  deficit  counter.  Therefore,  one 
can  combine  these  two  approaches:  predict  first,  and  then  cor¬ 
rect  if  not  accurate.  In  using  this  method,  the  PPLS  algorithm 
would  incorporate  some  of  the  principles  of  the  ERR  scheduler 
[12],  where  the  resource  requirements  are  not  assumed  to  be 
known  prior  to  the  allocation. 

C.  Discussions  on  Further  Extensions 

Note  that  in  our  study,  it  is  assumed  that  each  flow  has  a 
unique  weight  which  determines  its  relative  rightful  share  for 
each  resource.  If  instead,  for  each  flow,  a  different  weight  is 
associated  with  each  individual  resource,  the  premise  of  this 
work  can  still  be  applied.  The  only  difference  would  be  that 
when  defining  the  prime  resource  for  each  flow,  the  weight  for 
each  individual  resource  needs  to  be  taken  into  consideration. 
This  requires  an  additional  concept,  the  prime  weight,  defined 
as  the  weight  associated  with  the  prime  resource.  In  this  case, 
for  all  flows,  the  quantum  values  for  each  resource  in  the  PPLS 
algorithm  need  to  be  proportional  to  the  weights  corresponding 
to  that  resource. 
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Even  though  we  have  focused  only  on  the  processing  and 
bandwidth  resources,  the  FERA  principle  and  the  design  of  the 
PELS  algorithm  may  be  easily  extended  to  systems  with  addi¬ 
tional  essential  and  related  resources,  such  as  a  wireless  system 
where  processor,  link,  and  power  are  all  shared.  In  this  case,  for 
each  flow,  three  quanta  and  three  deficit  counters  are  needed. 
The  core  of  the  algorithm,  however,  remains  the  same,  i.e.,  a 
packet  from  a  flow  can  be  scheduled  only  if  all  three  deficit 
counters  of  the  flow  are  large  enough. 
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Appendix  I 
Proof  of  Theorem  1 

Lemma  1:  In  a  system  with  multiple  essential  and  related 
resources,  the  normalized  allocations  received  by  a  flow  i  are 
identical  under  two  allocation  policies  q  and  s  if  Af  =  Af. 

Proof:  Let  and  Bf  be  one  of  the  prime  resources  of 
flow  i  under  policies  q  and  s,  respectively.  We  have, 


Kb,  < 


K,bj 


A: 


i,B'’ 


-  ■ 


In  addition,  we  have  A'^.  „„  =  A®  since  A®  =  Af  Therefore 

1,0-  ^  ^ 

AJ  gs  <  A®gs  which  means  flow  i  receives  less  allocation  of 
resource  Bf  under  policy  q  than  under  policy  s.  Also,  since 
the  resources  are  essential,  flow  i  receives  no  more  allocation 
of  any  other  resource  including  B^  under  policy  q  than  under 
policy  s,  i.e.. 


This  means  B^  is  also  one  of  the  prime  resources  of  flow  i  under 
policy  s.  It  may  be  similarly  deduced  that  Bf  is  one  of  the  prime 
resources  of  flow  i  under  policy  q.  In  summary,  if  Af  =  Af,  the 
sets  of  prime  resources  of  flow  i  are  identical  under  policies  q 
and  s  if  =  Af,  and  the  allocations  of  these  prime  resources 
to  flow  i  are  also  identical  under  policies  q  and  s.  Since  the 
resources  are  related,  flow  i  receives  identical  allocations  of  all 
resources  from  both  policies  q  and  s.  ■ 

Now,  we  will  proceed  to  prove  Theorem  1  by  contradiction. 

Assume  that  in  the  considered  system,  when  applying  the 
FERA  principle,  two  different  policies  q  and  s  are  both  fair 
work-conserving  allocation  policies  corresponding  to  the  no¬ 
tion  of  fairness  T .  Denote  the  constraints  corresponding  to 
these  two  allocation  policies  by  C®  and  C®,  respectively. 

Note  that  for  these  two  allocation  policies  q  and  s,  the  vectors 
of  the  largest  normalized  allocations,  i.e.,  [Af]  and  [A®],  cannot 
be  equal.  This  is  because  if  that  is  the  case,  from  Lemma  1, 
the  allocated  amount  of  each  resource  for  each  flow  will  be  the 
same  under  policies  q  and  s,  and  policies  q  and  s  will  be  iden¬ 
tical. 

Since  the  system  under  consideration  remains  the  same,  we 
know  that  both  the  vector  of  the  largest  normalized  demands 
[T>i]  and  the  vector  of  the  flow  weights  are  the  same.  From 
the  definition  of  a  uniquely  deterministic  notion  of  fairness,  we 
know  that  C"^  f  C®  since,  otherwise,  the  two  vectors  of  largest 
normalized  allocations,  [Af]  and  [A®],  will  be  equal. 

Since  the  notion  of  fairness  T  is  non-decreasing,  from  the 
definition,  we  have  either  [Af]  ^  [A®]  or  [A®]  ^  [Af].  Without 
loss  of  generality,  we  assume  that  [Af]  ^  [A®],  i.e.. 


<A|,V2. 


(6) 


Therefore,  for  all  flows,  the  allocated  amount  of  each  resource 
under  policy  q  is  no  more  than  that  under  policy  s,  since  the 
resources  are  related  and  essential. 

In  addition,  there  must  exist  at  least  one  flow,  for  which  (6) 
is  not  an  equality.  In  other  words,  there  exists  at  least  one  flow, 
which  gets  more  resources  under  policy  s  than  under  policy  q. 
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Hence,  under  policy  s,  as  opposed  to  policy  q,  no  flow  gets 
less  allocation  for  any  resource,  and  at  least  one  flow  is  allo¬ 
cated  more  of  some  resources.  This  violates  the  assumption 
that  policy  q  is  work-conserving  and  completes  the  proof. 

Appendix  II 
Proof  of  Theorem  2 

Without  loss  of  generality,  we  assume  that  the  flow  weights 
are  normalized  in  such  a  way  that  the  smallest  of  the  weights 
assigned  to  a  flow  is  1. 

In  the  rest  of  this  proof,  we  will  limit  our  consideration  to 
the  situations  where  lines  19  and  22  in  Fig.  3  are  never  exe¬ 
cuted,  i.e.,  the  deficit  counters  of  any  flow  are  never  above  the 
thresholds.  The  reason  of  this  assumption  is  similar  to  the  one 
used  in  the  design  of  Idealized  Wireless  Fair  Queueing  (IWFQ), 
where  fairness  in  bandwidth  cannot  be  guaranteed  if  any  flow 
lags  more  than  the  maximum  lag  allowed  by  the  wireless  packet 
scheduler  [21]. 

Lemma  2:  In  an  execution  of  the  PPLS  strategy,  at  the  end 
of  each  round  k,  for  any  flow  i, 

1)  The  following  two  statements  are  always  satisfied; 

0  <  PDCiik)  <  TiidiX.{maxPDCi\ 
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.  {PDCAk)  LDCAk) 

\  Rp  '  Rl 
where  constants  a  and  (3  are  defined  as  follows: 


a  =  max  • 


'  nmx  {maxPDCi}  rnax  {maxLDCi}  ' 


Rp 


Rl 


^  .  ( Mp  Ml\ 

According  to  (5),  we  also  define  constant  7  as  follows. 


7  = 


minlPQJ 

Vi  Vi 


Rp 


Rl 


(7) 

(8) 

(9) 


Lemma  3:  During  an  execution  of  the  PPLS  strategy  over 
any  m  rounds,  for  any  flow  i. 


mwij  —  P  <  CRA"(m)  <  mwi^  +  a 


where  a,  P,  7  are  constants  defined  in  (7),  (8)  and  (9),  respec¬ 
tively. 

Proof:  Denote  by  SCPAi(fc)  the  cumulative  processor  al¬ 
location  of  flow  z  in  a  single  round  k.  From  the  algorithm  we 
have. 


0  <  LDCpk)  <  ma,yi{maxLDCi\\ 

Vi 


SCPA,(A:)  =  PQ,  +  PDCPk  -  1)  -  PDCPk). 


2)  At  least  one  of  the  following  statements  is  always  satis¬ 
fied: 

0  <  PDCifk)  <  Mp 
0  <  LDCpk)  <  Ml 

where  Mp  and  Mp  are,  respectively,  the  maximum  pro¬ 
cessing  cost  of  a  packet  and  the  maximum  link  cost  of  a 
packet. 

Proof:  First,  it  can  be  readily  verified  that  the  deficit  coun¬ 
ters  can  never  be  negative.  The  first  half  of  Lemma  2  can  be 
directly  derived  from  the  assumption  that  lines  19  and  22  are 
never  executed. 

Next  we  prove  the  second  half  of  Lemma  2  by  contradic¬ 
tion.  Assume  that  both  statements  are  not  true,  then  we  have 
PDCpk)  >  Mp  and  LDCpk)  >  Mp-  Note  that  at  this  mo¬ 
ment,  flow  i  still  has  packets  in  the  queue  waiting  to  be  sched¬ 
uled.  Otherwise,  both  deficit  counters  of  flow  z  should  be  reset 
to  0.  Consider  the  head-of-line  packet  of  flow  z,  say  p.  Ap¬ 
parently  its  processing  cost  is  no  more  than  Mp  and  its  link 
cost  is  no  more  than  Mp.  In  other  words,  its  processing  cost 
is  less  than  PDCpk)  and  its  link  cost  is  less  than  LDCpk), 
and  therefore,  based  on  the  PPLS  algorithm,  packet  p  should  be 
scheduled  in  round  k.  This  violates  the  assumption  that  packet 
p  is  the  head-of-line  packet  from  flow  z  at  the  end  of  round  k. 
This  completes  the  proof.  ■ 

Lemma  2  readily  leads  to  the  following  Corollary. 

Corollary  1:  In  an  execution  of  the  PPLS  strategy,  at  the 
end  of  each  round  k,  for  any  flow  z. 


This  leads  to 


CPAi(m)  =  ^SCPA,(fc) 


fc=i 


=  mPQ,  +  PDCP0)-PDCi{m) 


and 


CPA”  (to)  = 


=  TO 


CPAi(TO) 

Rp 

PQ^  ,  PDCPO)  -  PDCPm) 


Rp 


Rp 


From  (2)  we  have, 

PQ, 


min  {PQj} 


Rp  minizuj}  Rp 

Vj 


=  WiJ 


and  therefore 


CPA" (to)  =  mwiy  - 


PDCPO)  -  PDCPm) 
R^  ■ 


Since  both  P  DC  iff)  and  PDCfm)  are  non-negative, 

PDC^(m)  ^  PDCAO) 

mwij - - -  <  CPAj  (to)  <  mwij  + 


Rp 

Similarly  we  have 


Rp 


PDCfk)  LDCPk)\^ 
Rp  ’  Rl 


IILUJi  J  • 


Rl 


max 
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Applying  the  above  in  the  definition  of  normalized  cumula¬ 
tive  resource  allocation  leads  to  the  following: 


CRA"  (m)  <  mwi^  +  max 

CRA"(m)  >  mwij  —  min 


PDa{0)  LDC,{0)\ 
Rp  '  Rl  ) 
PDCi{m)  LDCi{m)  ^ 

j 


Applying  Corollary  1  into  the  above  inequalities  completes  the 
proof.  ■ 

Consider  a  certain  time  interval  [^1,^2),  during  which  all 
flows  remain  backlogged.  Consider  any  pair  of  flows  i  and  j. 
Assume  that  during  [^1,^2),  flow  i  receives  rrii  rounds  of  ser¬ 
vice  while  flow  j  receives  ruj  rounds  of  service.  Since  both 
flows  i  and  j  are  backlogged  during  time  interval  (^1,^2),  and 
the  scheduler  serves  the  flows  in  a  round-robin  fashion,  we  have 
\mi  —  m^l  <  1. 

Applying  Lemma  3  we  have 


CRA"(fi,f2) 

Wi 

CRA"(fi,f2) 

Wj 


a 

rui'y  -\ - 

Wi 


Therefore, 


CRA^(fi,f2)  _  CRA"(fi,f2) 

Wi  Wj 

a  (3 

<  [mi  —  mj)"f  -\ - 1 - 

Wi  Wj 

<  a  +  (3  +  J. 


Similarly  we  can  also  derive  that, 

CRA^"(fi,f2)  CRA^(fi,f2)  ^  , 

- ^ - <  a  +  (3  +  ^. 

Wj  Wi 

Since  flows  i  and  j  can  be  any  pair  of  flows,  based  on  the  defi¬ 
nition  of  the  normalized  fairness  measure  we  have. 


FM”(fi, ^2)  <  a  +  P  +  J. 

Note  that  a,  P  and  7  are  all  finite  constants.  Therefore, 
FM”(fi,  ^2)  is  bounded  by  a  finite  constant  over  any  time  in¬ 
terval  during  which  all  flows  are  backlogged,  i.e.,  the  fairness 
bound  FB"  exists  for  the  PPLS  strategy  and  it  is  finite.  This 
proves  the  statement  of  Theorem  2. 


